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contributions. The assessment should be done after a product has been 


University. The dimensions of the Eramework (product 
opment, istajbution,. implementation, utilization and effects) 
xplained in detail. The criteria are defined and enumerated. 

include systematic development, quality, user orientation, 
egic distribution, multiple channels, widespread distribution, 
ntial implementation, support systems, cost feasibility, 
ple patterns of use, time on task, integrated use, user 
faction, individual growth, organizational change, and societal 


‘in use for a period of time. The logical continuation of this 
‘research would be to quantify the dimensions and criteria. This would — 
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enabl 
poten 


é evaluators to compare R&D products using an overall impact 
tial score. (DWH) 
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A CONCEPTUAL FRAMEWORK FOR MEASURING R&D PRODUCT IMPACT 


William L. Hull, Kay A. Adams, and Debra D. Bragg . 
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Answering the que8tion "what difference did an educational innovation make 
in a practice setting?” is‘a difficult and time-consuming process. Yet the 
. ’ . a 


question will not go away. Scarcity of funds for educational research and 


development (R&D) places great importance on answers to this question. Spon- 


“sors of applied R&D often expect some evidence of successful change due to the 


innovation before continuing financial support. Given thesé expectations, 
. , ; 
more needs to be known about collecting impact data. 
ar : . “ ™~ 
The costs for conducting impact studies are high. Timing of the studies 
a : 


can be critical. Unfortunately, the timing of impact studies often corre- 


a’ 


sponds to the sponsor's funding calendar rather than the lapse time needed for 
an innovation es foster changes in an educational setting. Premature .evalua- 
tion of an R&D innovation can reveal little Chuiigis resulting in termination 
ofa project. Finally, rarely are conditions in practice settings amenable to 
the conduct of ingaet studies using experimental designs. Typically, random 
assignment of students to groups is not possibile, and natural control groups 
do not exist. Control groups seen axe replaced by documentation of naturally 
occurring events. Thus nthe study of impact can become a phenomenon based on 

_ perceptions rather than statistical evidence. 

~The objective of this research was to conceptualize the process of program 

improvement using sagaheee chit R&D projects. This process involved the 


development of a quality product, its distribution to primary user audiences,. 


effective implementation, and sustained use in a practice setting. If an 


evaluator can relate events associated with the’ quality diffusion and use of 


educational R&D products to impact, it should be possible to estimate the 
likelihood of change (impact) occurring from use of product. 


Staff ee the National Center for Research in Vocational Education at The 


. 


Ohio State University developed a framework to aid in estimating. the impact 
f rom R&D products. This. framework evolved over a five ae period, 1978- -82, 


as evaluations vere eanducted in a series of studies on vocational education 
: « s 


R&D innovations. The central data base for this framework was obtained 
through impact scadins of 28 selected state-developed and National Center 
products; both qualitative and quantative data were collected through case 

_ studies and wanrenen An scdendtve review of the diffusion literature revealed 
267 impact studies of educational innovations. After Wa ree was for- 
malized, it was réviewed by participants in the Fifth Nationwide, Vocational 


Education Dissemination and Utilization Conference, and revised. Since that 


« 


conference, the framework and criteria have been reviewed by nine experts in 
dissemination and evaluation, establishing its content validity; The fol- 
lowing assumptions were made in the development’ of’ the framework: 


@ Impact from R&D products is issue-orientated and site-specific; 
that is, the worth of a product for resolving a problem on site 
depends upon its relevance to the problem(s) on Enae site. 


e@ New ideas can be packaged in a transportable format for use in ~ 
diverse settings. f 
e The primary output from a funded R&D peg lat normally a rep Ap 
product, is the most usable vehicle for PRaeine effects of 
the project. , 
. } ‘ 
e R&D products can be used by persons not involved in their 
development. . 


e Acceptance and use of an innovation developed elsewhere is less 
expensive to an adopting site than developing P arnoVaEL Ons 


e Accountability is the driving force behind most impact studies. 

This .type of evaluation requires dteedtion to the specific situation in 
which the innovation is adopted (Patton 1982). This framework allows an 
Liatuatos to identify and categorize some of the most critical , general, 
Seen AONE needed for R&D imparts to occur in a’ practice setting. 

\ Despite the general nature of the APERE OEE it may not be “comprehensive 
enough to apply to Wi carting or to all R&D products equally. An evaluator 
can be situationally responsive by NAaDe this framework as a sranhing pede 
and expanding or eliminating criteria i" needed. An empirical test of this 
general medal: is necessary to determine to what degree the criteria, rejate to 


impact in various settings. 


ae R&D Impact Comceptual Framework 


Figure 1 depicts salient features of the cénceptual fronework. It con- 


tains dimensions of the framework and the criteria within each dimension. 


- 


' ‘First the dimensions will be explained, then the criteria. Thé dimensions 


~ 
PA - 


are: é i e 


mays rete ne ay ng 


Development. Impact begins with if depivccuants: of the seoainees The criteria 


of systematic development, high qvality, and user oriéntation can be attained 


a, 


_during development of ‘the dora Often, field test data can be used be 


basis for rating a product on ‘these criteria. ae a 


. 
4 


Distribution. The distribution criteria jshquld Pie OaERS the sam of 


the R&D product to primary audiences, increase the likelihood of tie product' 8 


acceptance, and generate support for its use. | : “ 


ne ee ee ee 
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DEVELOPMENT DISTRIBUTION IMPLEMENTATION UTILIZATION 
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Multiple 


Systematic Strategic Sequential 
Development Dissemination Implementation Patterns 
High Quality Multiple Support Integ rated 
- ‘ Channels Systems Utilization 
User Widespread Cost Time on Task 
Orientation Dissemination Feasibility 
FORMATIVE 


Figure 1. R&D Impact Criteria 
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Implementation. Implementation strategies determine the product's sotint 
of entry into an organization (e.g-, at the classroom level). Cost feasibil- 
ity studies and the need among practitioners for sipuee systems aid the 
timely implementation of R&D. 

Utilization. Various S deliee tua criteria can encourage the appropriate ¥ 
trial use of products, stimulate the integration of products into existing 
operations, and increase the chances of continued product use. 


R Effects. Product effects criteria should accurately describe changes in 


é 


‘individuals, organizations, or society attributed to BAR OE R&D innovatdions. 
The dimensions are somewhat sequential in nature (e.g., implementation of 
a product following distribution), heme: some recycling may occur among. the 
criteria. For example, a product may be high in quality, but not’ user orsen- 
tated, or it may have to be modified before it can be widely pete i 


However, in most case, these criteria will be met if the product is to have 


impact. 


P Dimensions Explained 


Paramount among these concepts is the, development of a high quality pro- 


" 


duct: A three-year study (Louis, Rosenblum, .and Molitor 1981) of .the National 


Institute of Education's research and development utilization program found 
product quailty to be particularly important in predicting the degree of 

impact. Product quality is considered a measure of the relevance of the pro- 
duct to the situation and the reer to: which it is a genuinely new way of . 
doing things. aia vee Center's Advisory Council reviewed criteria in the 
evaluation model proposed in this paper and rated the development criteria a 


the ‘very top of the list. Qualit¥ control of R&D products is difficult to 


achieve, however, and even more difficult to have accepted in the field as an . 


isipowtadt indicator of potential’ impact. As Klein (1978) points out: 
r . . susers are not systematic in their approach, and seldom use 
|. effectiveness criteria. My experience working with user groups, 

_ such as teachers, supports the notion that the users feel very 

unqualified to look at even summary reports on evaluation. 
(P. 119) : : 

’ Quality neat at R&D products’is an uphill battle for developers. Money . 
is scarce, and some people assume that money allocated for product development 
automatical ly results ae good product. What is needed is more evaluations 
of ‘products while they are being developed, with clear indicators of what the 
products can do for potential users. The great need for quality standards is 
documented in the Committee on Vocational Education Research and bevetouudak 
(1976) report and has been emphasized more vecentdy by Worthington (1981). —_. 

Widespread distribution of R&D products is implicit in the concept of com= 
snehan ate dissemination, as defined by the Dissemination Analysis Group (DAG) 
(1976). DAG's four-level definition of dissemination (i.e., spread, exchange, 
‘choice, and implementation) has been endorsed by, professionals in the field of 
sdicat inal dd wamctdeetan at the June 1977 Dissemination Forum. 

: The DAG conceptual definition of comprehensive dissemination forms an 
underlying eactondie tee assessment of impact, as captured in the conceptual 
framework ‘1 Figire 1. The first-level definition of spread resembles a one- 
way casting « knowledge similar to the idea of wides pread dissemination 
used in the impact conceptual framework. 

Another concept relating to impact from R&D products is effective imple- 
mentation of the product in diverse educational settings. Some people sub- 


scribe to the view that good products will sell themselves. But product 


developers do not always know how products will be used. As articulated in 


the DAG (1976) report, products are selected by users for program improvement 
taterventions, cae must be accompantes by support systems that 
involve anaes ae financial resource allocations. Product implementation 


and use must be éndorsed by those in authority for institutdionalization to 
/ A 


occur. wk 
Fullan, ada Pomfret (1977), in their review of curriculum implementation 
i t ‘ 


studies, identify che following factors influencing effective implementation: 


e characteristics of the uecuiae: such as its complexity or 
, difficulty of change required by the innovation; 
i. @ 2 
jo strategies of implementation dealing with resource support, Pe 2 
[ timing of the intervention, feedback mechanisms, and 
participation in the innovation process; 
Fi) 


j e characteristics of the adopting units, such as their demo- 
if graphics and ability to solve problems; and 


if e macro-sociopolitical Factors, such as incentive systems, Bie 
: role of evaluation, and pOseeSee> complexity. 


~ ' In an -insight ful article on incentives for innovation in the public 


schools, Pincus (1974) reviews bureaucratic factors supporting innovation and 
characteristics of institutional settings. He concludes by saying: 

- . dina diverse society. . .at any one time there will be a 

variety of standards. A major focus of R&D policy should be 

- . .experimentation and. . .incentives that encourage new 

patterns of institutional behavior. (p. 139) 

The possibility of an R&D product being modified in the process of 
becoming adopted by an institution is always very great. The Rand Corporation 
(Berman and McLaughlin 1978) conducted a study that included the Part D Exem- 
plary Programs authorized by the Vocational Education Amendments of 1968. The 
study showed effective projects to be characterized by mutual adaptation of 


both the product and the adoption site during the implementation process. The 


study also found professionalism to be a primary motivation when teachers 


undertake extra work. Clarity of objectives likewise had a major effect on 


-implementation. In addition, comprehensive projects were found to be no more 
effectively implemented than simple projects; in fact they were somewhat less 
likely to produce teacher change. These findings suggest the value of incre- 
mental goal setting when introducing an R&D product into an established edu- 


: * 
cational setting. 


Another concept related to impact from R&D products is sustained as of | 

the R&D product. Causal models constructed by Crandall et al. (1982) to 

ea 

explain findings from a sample of 146 schools in ten states show that teacher 
commitment and elapsed time (i.e., the length of time the teacher has been 
using the innovation) are significant predictors of change in suntdoes The 
centrality and importance of teacher commitment or ownership are underscored 
by consistent patterns in subsets of the data. Organizational change, on the 
other hand Syeflects the importance of the principal's management style and 
ieadeeuld ds The principal's ability to adopt a "take charge”, attitude pro- 
vides the d@nly route to institutionalfzatton of R&D-based innovations. 

Each dimension contains criteria that can be used to measure the likely 
impact of a product. The opt imal peasurement procedure would include multiple 
indicators for each ceriterian within the four dimensio#s. Kerlinger (1979) 
points to the Gives biitey of depending upon’ single indicators for meas ur e- 
ment. He describes measurement procedures as highly indirect, complex, and 
often difficult. When one considers the complex processes associated with 
impact, the need to give consideration and planning to measurement becomes 
doubly ecaliteueed, However, existing records may be used to collect data 9n 


dimensions of distribution, implementation and utilization. ‘ Also, fairly 


objective instrumentation has been developed to measure innovation use in 


. 


educational settings (Hall ahd Loucks 1977)’. Furthermore, where more sub- 
jective data are needed, rating scales with multiple items may. be used. For 5 


example, at the time of release, any given product can be rated on the cri- | 
; , ao 


terion of product quality: 


Product Quality 
LO Wr ane n nnn nnn High 


Scholarship i Wy See os 
Utility a a a a 

' * Communicability — 1 | 2 3 4 5 
Free of Biases . : oh oe 4 5 


- 


x 


A similar measurewent scale could be devised for other criteria within wick 
dimension where it is not possible to obtain more objective data. Figure l - 
classifies’ sixteen criteria by the five dimensions. A supporting rationale 
for the criteria and research findings related to development of the framework 
can be feed ae R&D Impact Criteria for Improving Vocational Education Pro- 
-grams (Hull, et al. 1983). The- following describes each seiterion oe a 


respective dimension. 4 


The Criteria Defined x * : 
Systematic development. . A systematic process should be followed in devel- 
oping R&D products. An ideal process would include conducting research and 
needs sienigonntl bank analysis; reviewing relevant knowl edge and practice; 
building a conceptual framework; sequencing development; conducting testing 
nd revision cycles; disseminating the product; implementing the product; and 


- 


evaluating the results. - 


~ 


Ii, 


High quality. Products should reflect scholarship, be useful, communicate 
clearly, be marketable, and be free of biases. Content should be accurate, 
up-to-date, focused on essentials, and complete. 


(User orieritation. Representatives of relevant audiences should be identi- 


fied and involved in designing, testing, and using innovations. Primary audi- | 


ences should receive priority in dissemination efforts. The resulting product. 


should contain practical information organized in an easy-to-use fomnat. 
| Strategic distribution. Cost-effective stintanien for distributing an R&D 
product should be devised on the basis of the characteristics of potential 
users, site-specific factors, and features of the product itself. Distribu- | 
tion should reach opinion leaders and influential organizations in‘the exter- 
nal environment. 
Multiple channels. More than one channel for conveying information about 


eohicti should be ‘eede? Communication should include mass wade (e-g-,. bro- 
chures sent nile by mail) and interpersonal channels (e.g-, technical assis- 
tance). Normally, information duplication and overlap are assets rather than 
liabilities during the distribution stage. 

Widespread distribution. Products should reach appropriate users. Thus, 
distribution ba dndiividuats in different roles, in diGnige bateiuae, and in 
many geographic areas may be necessary. Secondary distribution through work- 
shops, reprints, libraries, the ERIC system, -and so on should be eicnudawed: 

Sequential implementation. The introduction of products should be 
sequenced to meet the, needs and unique characteristics of an adopting site. 
Often potential users need to be introduced is wareante of the product eo. 


avoid total rejection of the intervention. 


° 
‘ 


10 
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—_ ee 
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Support bystens. PuppoRs systems iavaaels for encouraging the full use 
-of a “Product shout ve operational at the Rae of implementation. These sys- 
tems are of three RYRSSS personal resources (e<g., administrative PUARESERERE _ 
or site persdinel endorsenent), information resources (e.g., training in the 


use of BUPPOTE materials and stneetared>. and physical resources Ce. g-, dol- 


lats, supplies, and equi pment). 7 sp ele a : : ‘ 
ee Cost Serres Infomation suentavleg the product's resource require- — 
: ments sho allow quick and easy beclastee of the costs likely to be incurred 
. by an adopting unit. ee * 5 


Heletehs patterns ae ube, A product's use patterns vill vary according 8a 
the conditions, intensity level, frequency, and extent of use. The users' 
setting, role, and Aesouraphis chavactetincics create the conditions for dif- 
farent. types of use. Multiple patterns of use and sheGadaey dan of R&D by 
other than the primary user audience should be encouraged. — 


"s Time on task. -An R&D product should beysed £ suena anode and Logg - 


enough for its use to become an integral part-Of current practice. The audi+~ : 


ence's time in, actually using the product sitguld be maximized. 


: » . , ‘ . 7 ig : oe : 
. Integrated use. Use of a product cae te intensive and pervasive , 


throughout the organization. To accomplish chitsgoal , personal commitment is 
Senate within the organization at: atl levels to institutionalize the produtt . j 
into organizational routines. ae K . i : . : Pag 
‘User satisfaction. The R&D product and its smplenentation should meet 
users' expectations and senile ina positive user attitude toward the product. Ge 
' , User satisfaction may be indicated by product advocacy: or by Steative: 


adaptations. °° ; : ‘ 


,¥- 2 f rae . hs eS ‘ i a 
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Individual growth. Products should contribute to changes in an individ- 


ual's attitude, knowledge, or performance. : 


" Organizational change. R&D products should contribute to beneficial 


od 


ges in the user's organizational policy, programs, practices, or struc 


ture. Beneficial changes may also include cost and time savings over current 


practice. _ . _ 


Societal contributions: R&D products should contribute new and signifi- 
cant information with the potential to advance knowledge, improve current 


practice, or influence social systems. 


Summary , 
This framework was developed to_ aid evaluators in assessing the impact of 


R&D products. Ideally, this assessment would be done after the product has 


. 


been in-use for a period of time. ‘During these summative evaluations often it. | 


A 4, fe wie df 
is helpful to know how much the product has been used, what were the condi- 


tions, and che degree of product .implementation. This formative cieiunbicn 
Tatwsnnovides valuahix insight into the pervasivenéss of the impact. But in 
particular, the conceptual framework created in this pratace should help eval- 
uators estimate the likely impact of a R&D setae, A high quality product, 
widely distributed and: fully implemented, with-Bustained use, is likely to ' 


have more impact than one which fails to possess these dimensions. The pro- 


a 


ducts” progress toward these criteria can be assessed at various points in 


. 


time as appropriate. “In this way, a R&D product accepted and implemented in a 


local setting, acquires an “impact potential.” ’ 


4 


A logical next step. for further development of this framework would be to 


ee . a 


quantify the dimensions and the criteria. This would allow evaluators to com’ 


pare R&D products using an overall impact potential score. The score would 
oe ee Be : ; 


~— 


a 


~ 


\ 


= + 3 ; 
vary depending on the implementation setting. The systematic use of this 
framework for selecting and using sdacartoual R&D products should upgrade the 


program improvement process. The information from these assessments should 


help research administrators, project directors, and evaluators gain increased 


impact from scarce R&D dollars. 
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